TYCOON: Theoretical Framework and Software Tools for Multimodal Interfaces

W e define a modality as a process analyzing and producing chunks of information. For instance, a speech recognition modality analyses speech signals and produces the labels of recognized words. Several multimodal interfaces combining such modalities have already been developed (IMMI'95, CMC'95). To take benefit out of them so as to advance research and implementation of multimodal interfaces, coherent theoretical and software tools are needed. From the " theoretical " point of view, the development of multimodal interfaces addresses several issues (Maybury 91, Dowell 95): content selection (" what to say "), modality allocation (" which modality to say it "), modality realization (" how to say it in that modality ") and modality combination. This paper deals with the " modality combination " issue. A multimodal interface developer has to know how to combine modalities and why this combination may improve the interaction. Yet existing frameworks for human-computer interfaces do not answer these two questions. Instead, they deal with the relation between the modes (language or action), the channels (audio, visual or haptic), the media (speech, text or gesture) and the styles of interaction (command language, selection in a menu) (Frohlich 91). Other frameworks describe the specificities of each modality regarding information content (Bernsen 95) or the temporal and semantic relations between events detected on several modalities (Nigay & Coutaz 93, Kara-giannidis 95). From the " software tools " points of view, existing authoring tools enable only the multimedia developer to combine modalities on temporal and spatial dimensions. A common deficiency of these tools is the lack of support mechanisms for the design and implementation tasks (Väänänen 95). This paper describes our approach named TYCOON, which is based on the notion of TYpes and goals of COOperatioN between modalities. It covers both the theoretical and the software points of view. It is composed of a theoretical framework for studying multimodal interfaces, a specification language and a multimodal module integrating events detected by several modalities.